Great!

Here are some random notes:

MessagePack like API (splitting a parsing API to appending new data, parsing buffer and getting parsed data) is good.

MessagePack also uses feed but feed (a parser feeds data) may be a bit strange.

parser.consume(data), parser.refill(data) or something may be better. I used "consume" when I implemented a resumable parser in Apache Arrow ( apache/arrow#6804 ) but it uses callback style API and "consume" fills a buffer and parses the buffer. With this API, "consume" may be a bit strange too.

How about #value returns parsing data even when #parse returns false something like the following?

diff --git a/ext/json/ext/parser/parser.c b/ext/json/ext/parser/parser.c
index 749d594..f0731ac 100644
--- a/ext/json/ext/parser/parser.c
+++ b/ext/json/ext/parser/parser.c
@@ -2254,9 +2254,8 @@ static VALUE cResumableParser_parse(VALUE self)
 static VALUE cResumableParser_value(VALUE self)
 {
     JSON_ResumableParser *parser = cResumableParser_get(self);
-    json_frame *frame = json_frame_stack_peek(&parser->frames);
 
-    if (frame->phase == JSON_PHASE_DONE) {
+    if (parser->state.value_stack->head > 0) {
         return *rvalue_stack_peek(parser->state.value_stack, 1);
     } else {
         rb_raise(rb_eArgError, "no ready value"); // TODO: Figure out the best exception and message

@tompng shared an use case of resumable parser:

• It may be useful to process generative AI API response
• It's returned as a stream
• An application wants to display the response before a response isn't completed

For example:

1. Response: {"message": "This is a response. (not completed)
2. App: Show This is a response
3. Response: More messages. (not completed)
4. App: Append More messages. to its view
5. ...

The above diff doesn't satisfy this use case but we can use it for simpler case such as [1,. We can get [1] before we have rest data (something like , 2]).

If we also provide an API that returns not processed data something like the following, we may be able to cover the generative AI API response use case:

diff --git a/ext/json/ext/parser/parser.c b/ext/json/ext/parser/parser.c
index 749d594..d39ba11 100644
--- a/ext/json/ext/parser/parser.c
+++ b/ext/json/ext/parser/parser.c
@@ -2263,6 +2263,15 @@ static VALUE cResumableParser_value(VALUE self)
     }
 }
 
+static VALUE cResumableParser_rest(VALUE self)
+{
+    JSON_ResumableParser *parser = cResumableParser_get(self);
+
+    return rb_str_substr(parser->buffer,
+                         parser->state.cursor - parser->state.start,
+                         parser->state.end - parser->state.cursor);
+}
+
 void Init_parser(void)
 {
 #ifdef HAVE_RB_EXT_RACTOR_SAFE
@@ -2289,6 +2298,7 @@ void Init_parser(void)
     rb_define_method(cResumableParser, "feed", cResumableParser_feed, 1);
     rb_define_method(cResumableParser, "parse", cResumableParser_parse, 0);
     rb_define_method(cResumableParser, "value", cResumableParser_value, 0);
+    rb_define_method(cResumableParser, "rest", cResumableParser_rest, 0);
 
     CNaN = rb_const_get(mJSON, rb_intern("NaN"));
     rb_gc_register_mark_object(CNaN);

but feed (a parser feeds data) may be a bit strange.

Agreed. As I was working on this, I was thinking of simply renaming it to <<.

How about #value returns parsing data even when #parse returns false something like the following?

So your snippet wouldn't work, because the Hash and Array are only built once complete.

e.g. [1, 2, 3, the first element on the value_stack is 1, not [1, 2, 3]

However we could indeed build the partial object on demand. It's an interesting feature, I'll see about adding it, but I think it should either be a different method or an optional paramter, e.g. parser.value(partial: true), to avoid confusion.

an API that returns not processed data something like the following,

Interesting, wouldn't be hard indeed.